Panel Session What Does Dopamine Say: Clues from Computational Modeling

Background: Reinforcement learning models now play a central role in modern attempts to understand how the brain categorizes and values events traditionally framed by psychology as rewards and punishments. These models provide a way to design and interpret of reward expectancy experiments in humans across a wide range of rewarding dimensions. They also provide a connection to computational models of optimizing control, and hence connect the neurobiology of reward processing to simple forms of decision-making, even decision-making about social exchanges. A central signal in these computational accounts is the reward prediction error signal encoded by burst and pause responses in midbrain dopamine neurons. Numerous experiments have now provided strong evidence for the existence of such reward prediction error signals. Despite these successes, there is a missing piece to this story. The missing piece is a learning signal known as regret. By regret, we mean the difference between what ‘could have been obtained’ and what ‘actually was obtained’. Methods: We used several event related fMRI and hyperscanfMRI experiments to probe both reward prediction error signals and regret signals in humans subjects. We studied the reward prediction error signals using a simple conditioning paradigm where a light predicted the temporally consistent arrival of a juice squirt in the mouth of 25 human subjects. We also probed the existence of reward prediction error signals in another domain, social exchange, using a two-person economic exchange game (a trust game) and hyperscan-fMRI (n=96 subjects). A third experiment was carried out on to probe neural correlates of regret single human subjects carrying out an investment task (basically a gambling game). Results: All three experiments revealed strong correlates of these computational learning signals: reward prediction error and the regret signal. In both cases strong responses were observed in the ventral striatum, and in the case that choices were actually made by the subjects the prediction error signal activated ventral portions of the caudate nucleus consistent with previous reports using different tasks. The regret experiment showed exceptionally strong responses in the ventral putamen and also responses in Lateral Interparietal Sulcus area (LIP) that correlated with the value of the market fluctuation. In the trust experiment, we observe a signal in the ventral caudate that displays features of a reward prediction error signal. Discussion: These results address three major issues. (1) They show that reward prediction error signals possess detectable correlates in human brains using functional magnetic resonance imaging. (2) They show that reward prediction error signals show up in the ventral putamen when no action is required by the subject to obtain reward and the ventral caudate and putamen when an action is required. (3) Regret signals are treated by the brain as real losses and drive changes in behavior (behavioral results) and furthermore that this signals represent another form of learning signal, a counterfactual reward error signal, that has detectable neural correlates in the striatum, thus suggesting one physical substrate for the experience of regret. (4) Collectively, these results show the utility of using computational models to search for neural correlates of signals involved in reward learning and perturbed by disease. This approach provides a new direction to more traditional methods of searching for neural correlates of reasonable psychological categories. Dopamine Encodes a Quantiative Reward Prediction Error for For Reinforcement Learning Paul W Glimcher*, O’Dhaniel A Mullette-Gillman, Hannah M Bayer, Brian Lau and Robb Rutledge